The present invention relates to distributed messaging systems, and in particular to verifying message delivery in such systems.
U.S. Pub. 20040167932 is an example discussing distributed message queues. A distributed message queue system is a system with one of more nodes that produce data, one or more nodes that queue data, and one or more nodes that consume data. Messages can be delayed, lost, corrupted or repeated. Thus, there is a need for verification (e.g., as part of a quality review or audit). Verification has been used in various systems. An example of a data stream filter for verification/audit trail generation is described in U.S. Pub. 20080243979.
Apache Kafka is an open-source distributed publish-subscribe messaging system that is designed to be fast, scalable and durable. Kafka maintains feeds of messages on different topics. Producers write messages to topics and consumers read messages from topics. Since Kafka is a distributed system, topics are partitioned and replicated across multiple nodes. Kafka treats each topic partition as a log (ordered set of messages). Kafka retains all messages for a certain time, and consumers are responsible for tracking their position in each log. Kafka can thus support a large number of producers, a large number of consumers, and a large amount of data with little overhead.
LinkedIn developed a system for simple auditing of message streams using message counts in time windows. It would be desirable to have a more robust verification system to allow trouble-shooting and improving the efficiency of the system.